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Challenged by some of the inherent difficulties in teaching qualitative data analysis, three instructors 
created an interactive digital learning object entitled “Sleuthing the Layered Text: Investigating 
Coding.” In this paper we assess the effectiveness of that learning object as a tool for teaching 
qualitative coding. On the face of it, learning objects — a form of instructional technology that has 
been criticized for tending to be objectivist and content-driven — would appear to be ill-suited for 
teaching qualitative analysis, an open-ended, interpretive and subjective process. However, based on 
student evaluations from two very different undergraduate courses, we found that the learning object 
did prove to be an effective medium for teaching coding. We attribute this success to its design, 
which incorporates best practices of classroom instructors, and also to the integration of the learning 
object into our courses. Nevertheless, student feedback cautions us that the learning object is not a 
technological fix. The degree to which students valued the learning object over other methods of 
instruction was moderate, and some were leery about this form of digital technology substituting for 
classroom teaching, even though this was not our intention. 


Qualitative data analysis is notoriously difficult to 
teach and made even more difficult when taught in the 
learning environment of large, lecture-style 
undergraduate courses (Clark & Lang, 2002; Stalp & 
Grant, 2001). Conscious of this, three researchers who 
have all taught qualitative methods in various 
contexts — introductory methods courses, upper-year 
substantive courses and supervision — came together in 
response to a call for proposals for digital learning 
resources. We were looking for more effective ways to 
teach qualitative coding at all levels of undergraduate 
instruction, including large classes. The outcome of our 
collaboration was “Sleuthing the Layered Text: 
Investigating Coding,” a unique interactive learning 
object. 

Learning objects (LOs) are web-based instructional 
modules organized around specific learning objectives. 
Thus their purpose is to create self-contained, self- 
explanatory “virtual learning environments” 
(MacDonald et al., 2005, p. 81). Their design may 
incorporate various media including text, graphics, 
animation, audio and video, and any number of 
instructional elements such as tutorials, simulations, 
exercises, games, glossaries and quizzes. LOs are 
thought to enhance learning by being “engaging, 
interactive and fun” (MacDonald et al., p. 81). As web- 
based resources they are accessible at any time and can 
be used simultaneously by any number of learners. 
They are flexible, also, in allowing learners to navigate 
the content and activities at their own pace according to 
their learning needs. 

In many cases, learning objects provide instruction 
on a clearly-defined subject matter or technique; 
however, in creating ours, we set out to impart a 
complex skill — that of taking loosely organized textual 
data from interview transcripts or field notes and, 


through qualitative coding and analysis, deriving well- 
substantiated sociological argument. In “Sleuthing the 
Layered Text,” we likened qualitative coding to 
detective work; in this paper we make the LO itself the 
object of investigation as we evaluate its effectiveness 
in helping us teach coding. 

The Challenges of Teaching Qualitative Coding in 
Undergraduate Classes 

The challenges we faced as teachers of qualitative 
research to undergraduate students provided the 
impetus for us to create the LO. Four types of 
pedagogical challenges in particular brought us 
together, all of which relate to the complexity of 
qualitative methods and their unfamiliarity to many of 
our students who are better acquainted with quantitative 
methods. The first and perhaps the most basic 
pedagogical challenge is responding to the feelings of 
uncertainty and anxiety many students experience 
during the initial stages of coding about what the 
outcome of their analysis will be (Hein, 2004). 
Uncertainty is in the nature of qualitative coding. The 
process almost always delivers fresh insights, but 
researchers are rewarded only after an indeterminate 
period of slogging through data. This slogging typically 
unfolds slowly and iteratively, and sometimes it takes 
researchers down dead end paths. Just as some 
seasoned qualitative researchers admit to feeling 
daunted as they begin qualitative data analysis, students 
who are yet to experience ‘ah-ha’s’ from this process 
are all the more likely to feel trepidation when first 
learning to code (Clark & Lang, 2002; Connolly, 2003). 
Our challenge as instructors is to impart confidence in 
the process without minimizing the intellectual work 
involved. 
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A second set of pedagogical challenges relates to 
the readiness of our undergraduate students to embark 
on qualitative coding. Most students show up in our 
courses with an engrained sense of research as 
hypothesis testing. By the time they are asked to 
perform qualitative data analysis, most have already 
taken or are concurrently enrolled in a course in 
statistics. Their experience of research has entailed the 
progressive mastery of a set of standardized 
techniques that produce a single correct answer. As 
they are introduced to qualitative methods, they learn 
that the unitary epistemological framework of 
positivism, on which they may have relied in other 
courses, does not always hold sway in this research 
context. Epistemological debates abound in qualitative 
research, and to compound this complexity, the 
criteria for authorizing research findings as valid and 
reliable are also debated. As Hein (2004, p. 27) 
comments, the struggles of his graduate-level students 
to understand the diversity of approaches to 
qualitative inquiry are linked to their prior training in 
the quantitative research paradigm “and its criteria for 
evaluating research. These beliefs in quantitative 
research are often so ingrained that students are 
unaware of just how deeply they are held.” While 
methods courses and textbooks address key 
differences between qualitative and quantitative 
research paradigms, it is in the actual hands-on 
practice of qualitative analysis that students often 
struggle the most to grasp this new way of thinking 
about research (Raddon, Nault, & Scott, 2008). 

The interpretive and subjective nature of coding 
introduces a third pedagogical challenge because it 
means that coding must be guided by a researcher’s 
theoretical understandings and research questions. 
While there are a number of philosophies to coding, 
the general pattern requires that researchers read 
through a transcript multiple times, with each pass 
through the data taking the researcher into more 
abstract observations. One common way to approach 
this procedure is to distinguish descriptive, open, low- 
inference, or manifest codes from abstract, focused, 
axial, high-inference, or latent codes. Ultimately these 
focused codes lead to the development of over-arching 
themes. Sometimes researchers approach a data set 
with ‘sensitizing concepts,’ which are interpretive 
devices that help bridge the gap between evidence and 
theory, that is, between the concrete data and the more 
abstract, conceptual framework that emerges from the 
analysis (Bowen, 2006). Students who are still gaining 
a conceptual vocabulary and familiarity with social 
science debates will have a smaller repertoire of 
sensitizing concepts to use as starting points for 
qualitative analysis (Taylor & Bogdan, 1998, p. 140). 
Also, because of a lack of exposure to theoretical 
thinking, some students have difficulty grasping the 


distinction between a descriptive code and an abstract 
one (Blank, 2004). In the same way, when students 
move between concrete and abstract levels of analysis, 
they may have difficulty explaining their inferences. 
They may fear they are being too subjective in their 
interpretations, or they may be troubled by the 
multiple possibilities of coding. So, to teach coding is 
to teach theoretical thinking as well as the ability to 
handle complexity and ambiguity (Hein, 2004; 
Hopkinson & Hogg, 2004). 

Imparting confidence and a certain degree of 
independence is important, too, since even 
procedurally there is no single way to code (Esterberg, 
2002). Ideally we would want students to feel free to 
practice a variety of techniques. However, instructors 
are often faced with a dilemma; as soon as they model 
one coding technique, this is the one the students 
emulate. So, the pedagogical challenge of coding is to 
convey to students not only that there is not ‘one 
answer’ that they must discover but also that there is 
not ‘one procedure’ to arrive at that answer. 

While we hope that our students will be excited 
by the creativity of coding, the reality that we have 
experienced is that they need encouragement to endure 
the meticulous work and intellectual risk-taking that 
coding requires. They also need many opportunities 
for hands-on practice, to learn from others, and to 
receive feedback and direction. Indeed, this is how 
many of us have learned — as graduate students 
working with our own data with the support of 
supervisors, or by apprenticeship through working as 
research assistants on the projects of more senior 
researchers (Breuer & Schreier, 2007). 

Limitations of Available Resources and Strategies 

In developing strategies to address these 
challenges, the resources at our disposal frequently 
provide a limited form of help. For instance, the 
substantive qualitative research articles that we give 
students to read are of limited value in teaching 
qualitative coding because researchers seldom publish 
their data or write in detail about the process of data 
analysis (Barrett, 2007). While qualitative methods 
textbooks are widely available, we noted that these 
books emphasize qualitative research design and data 
collection much more than they do qualitative data 
analysis. To confirm this impression, we took a 
convenience sample of 10 qualitative methods 
textbooks that at least one of us has used in our 
teaching or as a reference book. We found that among 
all 10 books, a total of only 52 pages, or 2 percent, 
were dedicated to qualitative coding. A larger 
proportion of the books was dedicated to qualitative 
data analysis, but even this amounted to only 13 
percent of the books’ total page count (see Table 1). 
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Table 1 


Coverage of Coding in Qualitative Methods Textbooks 


Author 

Total pages 

Analysis pp 

Coding pp 

Berg (2007) 

366 

36 

11 

Bogdan & Knopp Biklen (2003) 

235 

37 

5 

Creswell (1998) 

229 

25 

3.5 

Esterberg (2002) 

256 

47 

5 

Glesne (2006) 

220 

25 

4 

Kirby, Greaves & Reid (2006) 

257 

35 

10 

Maykut & Morehouse (1994) 

163 

45 

5 

Schram (2003) 

134 

0 

0 

Shank (2002) 

208 

19 

5.5 

Warren & Karner (2007) 

294 

32 

3 

Total page count: 

2362 




Total page count dedicated to analysis: 301 (13% of total pages) 

Total page count dedicated to coding: 52 (17% of analysis and 2% of total pages) 


Before beginning work on the LO, we also 
considered using qualitative data analysis software for 
teaching coding. However, we doubted that the tutorials 
distributed by software companies could teach coding 
as both an analytical and a creative process, considering 
that, as Johnston (2006) points out, such tutorials are 
“specifically designed to teach software processes, not 
qualitative research methods per se” (p. 386). After 
looking at demonstration versions of two software 
packages, we concluded that they tend to give an 
impression of coding as a mechanical, linear, and 
algorithmic operation, whereas we wanted to convey 
that coding can be an open-ended activity for seeing 
meaning in data and something that may be done 
without specialized software. 

And so this is how we came to the project of 
creating our learning object. With a desire to teach 
coding experientially at the undergraduate level but 
faced with the difficulty of apprenticing an entire 
class or even managing students’ anxieties, we were 
looking for new pedagogical strategies and teaching 
tools to incorporate into our courses. But why create 
a learning object, what would it look like, and how 
effective could it be in helping us teach coding, 
given the challenges we have identified? We answer 
these questions in the following sections with a 
review of the literature on learning objects and on 
classroom-based strategies for teaching coding. We 
then describe the learning object we developed and 
present the results of quantitative and qualitative 
student evaluations of its learning value and 
functionality. These evaluations help us answer the 
following: 

1. Did students learn what we hoped they 
would learn about qualitative coding by 
using the learning object? 

2. Did students find value in the learning 
object as a teaching tool? 

3. Was the learning object a useful medium for 
teaching students about qualitative coding? 


Can a Learning Object Help? 

An integral aspect of the concept of LOs that 
appealed to us is their reusability (Barritt & Alderman, 
2004; D. A. Wiley, 2000). Their on-line storage makes 
them retrievable for use with successive groups of 
learners in the same course, and their focus on discrete 
learning objectives makes them resources for use in 
different courses in which those learning objectives are 
relevant, such as our qualitative research methods 
courses and subject-area courses requiring qualitative 
analysis located in different departments and 
disciplines. To facilitate reuse, each LO is encoded with 
information about its characteristics (title, author, 
format, and so on). International standards prescribe the 
categories for this descriptive information, known as 
“metadata,” to ensure that LOs may be identified 
through searches of on-line repositories. Once located, 
LOs may be made available to learners through course 
management system software such as Blackboard, 
WebCT, and Sakai. 

The potential utility of LOs in multiple educational 
and training contexts has generated a sizable industry 
around their creation and reuse, along with a 
burgeoning literature, mostly technical and promotional 
in orientation. Most champions of instructional 
technology tout LOs as cost-effective vehicles for 
creative and effective learning (see Bennett & McGee, 
2005), but a handful of authors resist the notion that 
LOs can be a technological fix for the pedagogical 
challenges of higher education and the fiscal constraints 
of academic institutions. These critics raise a number of 
points, ranging from concerns deriving from the 
funding of LOs by the US Department of Defense for 
the purpose of ‘any time/ any where’ military training, to 
doubts that LOs will meet the promise of reusability 
(Friesen, 2004; Parish, 2004). While we shared many of 
these concerns at the outset of this project, within this 
paper our focus is on the suitability of LOs for the 
challenges of teaching qualitative analysis. We were 
drawn to the concept of an LO but had to ask, is there 
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anything inherent in the LO model that would limit 
their effectiveness for teaching qualitative coding? 

This question arose for us out of skepticism 
triggered initially by the very term Teaming object,’ 
which, as we later learned, originated in a specialized 
area of software programming that involves generating 
standardized code to be borrowed by other 
programmers. Debate about the appropriate name for 
and definition of LOs suffuses the literature. We would 
agree with Ip, Morrison, and Currie (2001) and Friesen 
(2004) that the reference to object-oriented software 
design in the term ‘learning object’ is meaningless to 
educators and reveals the dominance of technologists in 
the propagation of the concept. More troublesome for 
us, the word ‘object’ in learning object connotes an 
objectivist orientation to knowledge and learning. As 
Parish (2004) argues, describing representations of 
knowledge as objects “inclines people to believe that 
knowledge itself is objective, rather than subjective, 
tacit, and dynamic” (p. 59). Of course, an objectivist 
understanding of knowledge is a cornerstone of the 
positivist paradigm that tends to dominate our students’ 
ways of thinking (Hein, 2004). As qualitative methods 
instructors, this is the kind of default assumption that 
we are attempting to shift to make space for alternative 
conceptions. 

We had some initial worries, too, that creating a 
learning object could be an exercise in packaging the 
complexities of coding into an overly simplified 
consumable experience for our students. The critics 
have pointed out that an excessive focus on delivering 
content and a tendency toward abstraction from 
context have made the majority of LOs reductionists, 
didactic instruments that perform little more than 
“information shoveling” (Ip et ah, 2001; Parish, 2004; 
D. Wiley et ah, 2004). As these commentators 
caution, LOs are more often conceived as “content 
chunks or information containers” than as resources 
for use in active learning contexts (D. Wiley et ah). 

In the end we were not deterred by the term 
‘learning object,’ and we were reassured by the 
instructional technologist at our institution (whose 
job it was to promote LOs to faculty and to support 
us with his technical design expertise) that the 
format was amenable to the kind of contextualized, 
interactive learning situation we were hoping to 
create. Nonetheless, the unusual nature of our 
learning object, because of its focus on teaching 
analytic skills rather than conveying specific 
content, sometimes offered new challenges to the 
technological support team that built it. We insisted 
on a complex design, knowing that the success of 
our project would depend on how well the LO could 
approximate the best elements of classroom 
teaching. 


Classroom-based Strategies for Teaching 
Qualitative Coding 

Although the pedagogical literature seems to lack 
well-documented strategies for teaching and modeling 
qualitative coding, we were able to identify five articles 
by both graduate and undergraduate instructors who 
teach coding as the fundamental process in qualitative 
data analysis and who present clear strategies to address 
a range of the challenges to coding. These documented 
strategies are all classroom based, as distinct from 
digital, and they suggested to us a set of best-practices 
which we attempted to incorporate into the design of 
the LO. 

Most significantly, all five articles address the 
challenge of demonstrating that coding may result in 
multiple interpretations of the data. These instructors 
did so mainly by leading students through an exercise 
of coding a common text or textual data set and then 
using class discussion, often in conjunction with small 
group work, in order to allow students to articulate and 
compare their interpretations. Instructors were creative 
in how they identified or generated these shared texts. 
In Roger Clark’s undergraduate methods course, for 
example, the data set consisted of students’ written 
commentaries about an image of Sojourner Truth 
(Clark & Lang, 2002). All the student commentaries 
were collated anonymously and distributed to the class 
for coding. 

Instructors tended to use a combination of 
independent activity, small group work, and larger 
discussion formats so that students could compare and 
reflect upon their choices and brainstorm new ideas 
together. As a result of the group discussion in Clark’s 
classes, for example, students became aware that “their 
decisions represented only a few out of an infinite 
variety that could have been made” (p. 352). Similarly, 
group work turned out to be the key experiential 
process in a coding workshop led by Philip Burnard 
because students discovered that the coding categories 
they had identified individually differed from those of 
their peers. Burnard writes, “Learning [coding] as a 
group reinforces the subjective nature of qualitative 
data analysis... There is rarely one right way to analyze 
textual data, unless a very mechanical form of content 
analysis is used in which words and phrases are 
counted” (p. 281). Accordingly, we aimed to design our 
LO so that it could support a variety of learning 
situations, especially group discussions where students 
compared their coding results. 

All five of the instructors whose exercises we 
reviewed also sought to help students make stronger 
connections between theory and data. In Stalp and 
Grant’s (2001) undergraduate course on field methods, 
for example, students were asked to apply a previously 
developed coding scheme (set of analytical categories) 
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to a set of personal ads that had been distributed to each 
student. In discussion of the exercise, students learned that 
they had to provide a theoretical rationale for their 
classification choices. Similarly, we designed our LO to 
model how researchers engage in a dialogue of theory and 
data, and to give students practice in doing the same. 

Another commonality among the exercises that we 
sought to emulate in the LO was their reinforcement that 
coding is a multi-stage process. In three of the courses that 
were reported on, students first generated a coding scheme 
from close examination of the data set and then in the next 
stage practiced applying the scheme to the data. Philip 
Burnard’s postgraduate nursing students, for example, first 
worked in pairs to code a common 3 to 4 page interview 
transcript. Then as a class they discussed the various 
outcomes before agreeing upon a common coding scheme. 
The final stage of the exercise was to apply this coding 
scheme to the same text by marking up the transcript with 
coloured pens (Bumard, 1996). A sociology graduate 
course taught by Grant Blank followed a broadly similar 
process but required students to work independently on a 
more rigorous set of exercises with a larger data set 
(Blank, 2004). 

In these exercises, as in our LO, instructors created 
opportunities for hands-on practice in order to teach the 
core lesson that “in qualitative analysis, phenomenon are 
not always clearly bounded and classifiable into mutually 
exclusive categories in the manner presumed in many 
quantitative approaches. Rather, phenomena can be multi- 
layered and have variable meanings” (Stalp & Grant, 
2001, p. 21 1). What seems to be lacking in four of the five 
exercises, however, is a way to convey to students the 
diversity of methodological and theoretical perspectives 
that inform researchers’ approaches to qualitative data 
analysis. 

Only one course, a small, upper year undergraduate 
course reported on by Karen Harlos and her colleagues, 
provides an instructional format that fills this gap (Harlos, 
Mallon, Stablein, & Jones, 2003). Harlos invited a panel of 
three guest researchers to each model a distinct 
epistemological approach to the reading of a three-page 
transcript. The three guest researchers represented 
interpretive and critical perspectives, but the authors note 
that the panel could have been expanded to include a 
realist, functionalist, phenomenologist, or representative of 
any other intellectual tradition that employs qualitative 
methods. Each of the three guests already had introduced 
themselves to the undergraduate class in separate guest 
lectures about their research programs, which centered 
around interpretive interviewing for the first researcher, 
grounded theory analysis for the second, and critical 
discourse analysis for the third. On the day of the panel, 
the three researchers and the students received the three 
page transcript excerpt for the first time. The nearly two 
hour session engaged panel members and students in two 
successive rounds of close reading of the transcript and 


discussion of their initial thoughts, tentative insights, 
questions about the data and ideas for exploring further 
connections. The session succeeded in demystifying the 
process of qualitative analysis as the researchers talked 
through their feelings about the data and made explicit 
their reasons for reacting as they did. As Harlos et al 
report, the session also left students “variously excited, 
intrigued, irritated, and concerned about the multiple 
possible readings of the text” (p. 314) — a similar outcome 
to the other exercises. Harlos’ students also practiced 
coding along with the panelists and had the opportunity to 
compare notes as they questioned the more experienced 
researchers. 

In the next section we show how “Sleuthing the 
Layered Text” incorporates the main features of these 
documented teaching strategies, particularly the provision 
of a common excerpt from a transcript, the opportunity for 
students to practice two stages of coding on their own, 
and, following Harlos et al, the comparison of different 
approaches to coding. 

A New Strategy: The Development of the Learning 
Object 

We designed “Sleuthing the Layered Text” to achieve 
five learning outcomes, which were, in ascending levels of 
complexity: to understand coding as interpretive and 
subjective, to recognize multiple coding philosophies and 
techniques, to distinguish between open and focused types 
of codes, to experience coding as a multistage practice, 
and to perform both open and focused coding given the 
opportunity for hands-on practice. Built in the technical 
format of Flash, the LO features audio and video media 
and takes about 30 minutes to complete. It can be the basis 
for group discussion, as the instructional literature would 
recommend, and students can also use it independently in 
their own time. 

Figure 1 

Learning Object Entrance Page 



After a welcome message and a one-paragraph 
explanation of qualitative coding, it begins by introducing 
students to the data — a fictitious transcript of an interview 
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with a fifteen year old high school student who is involved 
in her school’s environmentalism club. The transcript 
excerpt that is presented is said to be taken from one of 
thirty interviews with young environmentalists. 


What does environmentalism mean to you? 

And how did you get involved in environmentalism? 



Figure 2 

Interview Transcript Excerpt 

Lana's Interview 

You may click on AUDIO OFF or AUDIO ON to tum off or tum on narrations. 
Click on the questions below to reveal Lana's responses. 

Click here to open / download the full transcript segment 


Do you see yourself still being an environmentalist 


Are you parents supportive of your environmental 


The transcript excerpt is to be coded by three very 
different researcher personas, the three ‘sleuths’ who are 
introduced next. Each persona presents his or her 
theoretical interests, which primes students to read the 
transcript through different theoretical lenses and to notice 
how the salience of the text changes accordingly. 


Figure 3 

Discourse Desiree 



Hi there. I’m interested in the 
language that we use. What 
assumptions are embedded in 
our language and how does 
our language shape how we 
see the world? What are 
some political consequences 
of using certain words instead 
of others and who benefits? 
Specifically, I'm interested in 
how we talk about nature and 
how we naturalize social 
behaviour. 


Discourse Desiree 

M3 


Meet The Sleuths 

Discourse Desiree 



1 jlIKSD h 

click here to download the video narrative 




l 


Figure 5 
Political Economy Paula 


Hey folks. My studies are in 
feminist political economy. 

I’m interested in how people’s 
lives are shaped by their 
location in relations of gender, 
class, race, and other social 
structures. I want to 
understand how social 
inequalities are peipetuated 
and how marginalized people 
can effectively struggle 


Meet The Sleuths 

Political Economy Paula 


oppression. 


Political Economy Paula 

'USEgD*] 
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click here to download the video narrative 
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Through short movies focused on the marking of 
text, students watch and listen as the three sleuths 
proceed to code the same text in order to model their 
different approaches. While coding the transcript 
excerpt in stages, the researcher-sleuths demonstrate 
how to focus and refine their coding categories. They 
explain their method of working with the data and 
convey how they arrive at higher levels of abstraction 
with each reading of the data. Each researcher is 
distinguished by his or her distinctive hand-written 
notes in the margins and by various techniques for 
recording or applying codes, such as the use of color 
highlighting, underlining, a code book, memos, or a 
master list of codes. In this way, the LO visually 
demonstrates the basic lessons that the same text can 
support distinct sets of interpretations, that analytical 
acuity develops through repeated close reading and 
marking of a text, and that practical techniques for 
coding may vary. 

After viewing the demonstrations, students are then 
presented with another segment of the transcript for 
their own practice. In this part of the module, students 
have the opportunity to do two levels of coding, which 
we call open and focused, and to print their results. 


I 


ft 


:n 


in 


Hello. My research interests 
relate to people's participation 
in social movements and 
activism. I'm particularly 
interested in how people 
become committed to social 
causes. I want to understand 
what motivates people to 
• become socially active, the 
.4-. challenges they face as 
’ ^-activists, and how activists 
'-■'negotiate their way through 
these challenges. Underlying 
my focus on social movements 
is a theoretical interest in the 
interplay between behaviour 
and identity. 


Figure 4 

Social Movement Sam 

Meet The Sleuths 


Social Movement Sam 

[GBsTl] 


Social Movement Sam 
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click here to download the video narrative 
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Figure 6 
Focused Coding Practice 

_ . ,, ., Need help? Click on one of the sleuths for a c 

Focused Coding 



of the sleuths for a coding clue. 

000(j§B 

Transcript Your Focused Coding and Memos 


I*] Interviewer: Are your parents supportive 
of your environmental activism? 

Line 1) No, they're not, They think I'm 
going to get arrested or something, and 

Line 2) they think that I'm going to get 
sick or that I'm going to not like them 
because of being a vegetarian. 


Line 4) Sometimes I can persuade 
them to understand why I'm concerned 
about these things. 

Line 5) Like, I say things to them like 
'look at what’s happening at that WTO 
conference dad, looks who’s there and 
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During focused coding they are prompted towards 
greater abstraction through the help of optional hints 
from each sleuth that they may select if stuck. Such 
hints involve questions that encourage more critical and 
abstract engagement with specific lines in the text. At 
any point they can use arrow buttons to navigate 
backwards or forwards and they can open up a glossary 
of basic terms such as ‘code,’ ‘memo,’ ‘qualitative 
research,’ and so on. After the practice component, 
students proceed to a yes/no quiz that allows them to 
self-assess their comprehension of the main ideas and 
vocabulary that were conveyed throughout the LO. 

Evaluation of “Sleuthing the Layered Text: 
Investigating Coding” 

As mentioned, learning objects are typically 
housed in searchable on-line repositories so they can 
be available to instructors in various institutional 
settings and disciplines. Before our own LO was 
linked to its repositories, we conducted quantitative 
and qualitative evaluations of its effectiveness with 
two different groups of students. The purpose of this 
evaluation was not only to assess learning outcomes 
(Did students learn what we hoped they would learn 
about qualitative coding using the learning object?) 
but also to assess the value of the LO itself as a 
learning tool and as a medium for teaching about 
qualitative coding. For such an evaluation, comparison 
of results with a control group that did not use the LO 
would have been ideal, but because each instructor 
had introduced other changes to her courses and/or 
coding assignments, an appropriate control group did 
not exist. Our next best evaluation method was student 
self-reporting. For this we used a generic learning 
object evaluation tool that had been developed by a 
group of researchers at the Centre for Learning and 
Teaching through Technology at the University of 
Waterloo (Schoner, Buzza, Flarrigan, & Strampel, 
2005). The questionnaire provides quantitative data on 
four variables: a learning object’s learning value for 
students, the ‘value-added’ it contributes to other 
components of the course such as lectures and 
readings, the usability of its design, and its technical 
functionality. The questionnaire also provides for 
qualitative evaluation through three open-ended 
questions: 

1. What were the main strengths of the 
Qualitative Coding learning object? 

2. What were the main weaknesses of the 
Qualitative Coding learning object? 

3. In your estimation, is it desirable to use 
technology to support teaching and learning in 
campus-based university courses? Why or 
why not? 


This evaluation instrument had been used already 
with at least nine different LOs with different 
instructors in at least that many courses at different 
universities. Vivian Schoner and colleagues selected 
four of these studies in order to demonstrate the 
questionnaire’s flexibility in assessing a range of LOs 
across disciplinary and course contexts (immunology, 
human nutrition, particle physics, and introductory 
chemistry). In addition they compared questionnaire 
results with instructor interviews in order to confirm the 
face validity of the questionnaire as an evaluation tool. 

We assessed “Sleuthing the Layered Text: 
Investigating Coding” in two very different courses: a 
high enrolment, required qualitative methods course 
(Year 2 Methods) in a Sociology department (n=66) 
and a smaller, elective, advanced, content-related 
seminar course (Year 4 Seminar) in a department of 
Child and Youth Studies (n=17). The Year 2 Methods 
students were given the LO to work with in a 50 minute 
session in a computer lab. Five groups of approximately 
15 students used the LO at individual computers in the 
lab with the instructor or a teaching assistant present to 
answer questions about both the LO and how the 
instruction applied to their upcoming coding 
assignment based on data from another course project. 
The last screen of the LO prompted students to 
complete the voluntary on-line questionnaire that was 
appended to the module. Sixty-six students submitted 
their answers electronically to a questionnaire database. 

In the Year 4 Seminar class, 17 students completed 
the questionnaire on paper in class, having used the LO 
outside of class as a prelude to completing a coding 
assignment with another data set. In this case, in order 
to see that students actually had used the LO, the 
instructor required them to turn in their LO-generated 
practice coding at the same time they submitted their 
assignment. This practice coding was not evaluated. In 
both courses, the instructors timed the introduction of 
the LO to follow their own lectures and assigned 
readings on qualitative coding. 

Because the questionnaire was in two parts, with 
the open-ended qualitative questions following a series 
of Likert-type quantitative questions, we analyzed 
results for each part separately. 

Results of the quantitative evaluation. The 
results of the quantitative section of the questionnaire 
are reported in Appendices A, B, and C. Appendix A 
presents the item by item results for the Year 2 Methods 
class, and Appendix B does the same for the Year 4 
Seminar class. Appendix C compares the results for 
both classes, including the percentage of students who 
‘strongly agree’ and ‘agree’ with each questionnaire 
item. Appendix C also includes the mean scores for 
each item, the grand means for the four main variables, 
and the results of two-way t-tests of significant 
differences between the mean scores of the two classes. 
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Overall, the results of the questionnaire show that a 
majority of students in both classes had a positive 
experience of the LO across all indicators. Further, the 
evaluations of the LO by the two classes were highly 
congruent, which suggests that the LO can be 
successfully integrated into different learning 
environments. More specific findings related to the four 
main variables of the questionnaire are presented 
below. 

Learning value. Items related to the learning value 
of the LO received high scores. Averaging these items 
indicates that approximately 92 percent of Year 2 
Methods students and 84 percent of Year 4 Seminar 
students ‘strongly agree’ or ‘agree’ that the LO helped 
them learn qualitative coding. Both groups of students 
showed the strongest level of agreement with items that 
related to the LO providing them with an alternative 
way to learn the course material, including that the LO 
helped them “visualize the concept better’’ and learn the 
material “in a new way” and “at my own pace.” 

In comparing the scores of the Year 2 Methods and 
Year 4 Seminar students, no significant differences 
were found in the ratings for any of the questionnaire 
items, although the percentage of agreement and overall 
means for all items were higher for the Year 2 students, 
suggesting that the Year 2 Methods students found the 
LO to be more valuable to their learning than the Year 
4 Seminar students. The widest differences were for the 
items related to understanding the concepts and 
integration of the LO with other course components. 
We can see how this difference would result. For 
example, the Year 2 Methods course was centrally 
focused on learning the techniques and concepts of 
qualitative methods, whereas in the Year 4 Seminar 
course the coding exercise was brought in as a tool to 
help students analyze data that they were to then use in 
the preparation of a comprehensive paper. Also, these 
students in Year 4 had already been exposed to 
qualitative research in a previous methods course. 
Given their background knowledge, they may not have 
had as much to learn from the LO as the Year 2 
students. 

Value-added. Compared to the other three 
variables, in both classes the scores were lowest for the 
variable related to how the learning object ‘added 
value’ to other forms of instruction. Among the Year 4 
Seminar students, fewer than two thirds “strongly 
agreed” or “agreed” that the LO was a valuable 
supplement to other forms of instruction. The items for 
this variable also received the lowest mean scores. It 
should be noted that very few disagreed and none 
“strongly disagreed” that the LO added value, but a 
sizable minority in both classes expressed ambivalence 
about the advantages of the LO in comparison to 
lectures and assigned readings. In the questionnaire, 
value-added was conceptualized in terms of whether the 


LO was able to replace lecture or textbook learning, 
reduced the amount of time spent studying notes, and 
exposed learners to situations that could not be done in 
the classroom or through textbook reading. Of the three 
items, highest ratings of agreement were for the item 
related to exposure to new situations, which 
corresponds with the high ratings of the learning objects 
as an alternative way to learn the course material. 
However, the low ratings of agreement for both the 
potential of the learning object to replace in-class or 
textbook learning and reduce the amount of time spent 
grasping new material are important to note, as it 
suggests that for these students, the LO did not provide 
an easier or faster learning avenue. The qualitative 
results (below) help us interpret this finding. 

Usability. Another variable that received high 
ratings and the highest mean scores for both classes was 
‘usability.’ On average, over 90 percent of students 
considered the LO easy to use, with no significant 
differences found between the two classes on the 
questionnaire items related to their ability to navigate 
through the module, understand instructions, and follow 
the flow of the material. We are pleased by these results 
as we felt that we had developed a LO that was quite 
complex and ambitious, yet students reported that the 
content was well sequenced and intuitive. For us, it 
showed that it was possible to incorporate a linear and 
stepwise learning module into an interpretive and 
subjective process. 

Technological functioning. That said, the variable 
‘technical functioning’ received mixed scores from 
students, with the students from the Year 4 Seminar 
class rating two of the items significantly lower than the 
Year 2 Methods students. While almost all students in 
both classes found the technology suitable in terms of 
their skill levels, the Year 4 Seminar students reported 
experiencing technical difficulties and hardware and 
software problems with the program. These ratings are 
in part due to a problem with the printing function that 
arose for the Year 4 Seminar students who tried to 
complete the LO from their home computers and who 
were expected to print something and hand it in. The 
printing problem has since been corrected, but we are 
left to wonder whether the questionnaire results should 
be taken as anomalous or whether such ‘glitches’ are 
actually endemic to interactive instructional 
technologies. 

Results of the qualitative evaluation. We coded 
and reflected upon students’ open-ended comments in 
order to identify patterns in responses that would 
embellish the numerical ratings and convey in more 
detail the way they felt the LO helped or did not help 
them learn. The qualitative comments were particularly 
useful in conveying student attitudes towards the LO as 
a medium of instruction. Coding of the qualitative 
comments involved several passes through nine pages 
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of qualitative comments from the second year course 
and three pages from the fourth year course, affixing 
descriptive codes to each comment and then 
distinguishing between the most common codes and the 
‘negative cases.’ 

Supporting the quantitative data, students identified 
a number of strengths in the LO, with a majority of 
student comments falling into one of five codes. Year 2 
students, in descending order of frequency, suggested 
that the LO was easy to use (15), visually engaging 
(10), an opportunity to practice coding (5), successful at 
providing different perspectives (4), and interactive (2). 
Year 4 students similarly felt that the LO was easy to 
use (6), visually engaging (3), successful at providing 
different perspectives (3), an opportunity to practice 
coding (2), and interactive (1). As we have discussed, it 
is particularly difficult to teach coding without 
providing students with a singular template which they 
all then follow. Therefore one of our main goals in 
producing the LO was to provide an avenue for 
communicating the multiple approaches to coding that 
can be taken. While this was not identified as frequently 
as the more practical elements of the LO, seven 
students noted this advantage in comments such as: 

We got to see what three researchers’ coding 
looked like. We could see how they underlined, or 
highlighted and we got to see how they go about 
the process of coding. It was also very interactive 
and I like the fact that we could work at our own 
pace. (Year 2 Methods) 

Easy to navigate, provided a number of 
perspectives/techniques for qualitative coding. 
(Year 4 Seminar) 

Interactive, gave good visual examples of coding, 
showed different perspectives and how they affect 
coding. (Year 2 Methods) 

I thought it was very helpful because it was a 
visual way of learning the coding process, and it 
took you through it step by step. I also think it was 
useful that three different perspectives to coding 
were shown. (Year 2 Methods) 

Many other students commented more generally on 
how the LO provided a helpful opportunity to 
practice coding, particularly as they found the 
technology to be familiar, flexible and, for the most 
part, easy to use. 

The program was separated into very logical, 
very in-depth steps that were really easy to 
follow and understand. Also, the “sleuths” were 


very efficient in bringing forth relevant 
information, tips and pointers. (Year 2 Methods) 
The second year students were particularly pleased that 
the LO provided them with a visual representation of 
coding: 

I enjoyed “watching” as the researchers coded the 
transcript. Watching the coding as well as listening 
to their thoughts was very helpful (Year 2 
Methods). 

Overall, a majority of second year students commented 
either on the ease of use of the LO or the appeal of the 
LO’s visual dimension. 

Students also noted various weaknesses to the LO. 
Eight of the Year 2 students felt that the room they 
were coding in was too noisy because they were in a lab 
with many other students who were listening to the LO 
at the same time. This was their most significant 
complaint, although another five also encountered 
technical problems despite the common response that 
the technology was easy to use. Others found the LO 
boring/repetitious/long (6), even though it only took 
half an hour to complete. 

[A main weakness was] when the ‘sleuths’ ... 
seemed to ramble on about the same 
things over and over. (Year 2 Methods) 

...is very tedious work. (Year 2 Methods) 

The Year 4 students’ noted weaknesses were almost 
entirely related to technical problems — they provided 
twelve comments to this effect. Not only were the 
fourth year respondents frustrated that the print 
function did not work properly, but others found that 
they lost their work when they hit ‘back space’ or 
found the font too small during several of the coding 
presentations. 

Finally, in terms of weaknesses, six students from 
the second year course remained confused about 
focused coding, despite the assistance of the LO, 
highlighting the on-going challenge of teaching 
coding and the importance of supplementing any 
technology with in-class teaching. 

I found it somewhat confusing because I did not 
entirely understand the focused coding aspect 
(Year 2 Methods) 

It perhaps didn’t focus enough on explaining the 
differences between open and focused coding. 
Maybe a little more insight into these is necessary 
(Year 2 Methods). 
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These comments suggest to us that a LO is best 
supplemented by in-class teaching and more personal 
guidance. 

Finally, when asked about the overall value of 
using technology such as learning objects as part of 
campus-based courses, Year 2 students again cited the 
value of hands on/practice (9), how it fits well with the 
role of technology in their lives (8), that it is a new 
approach (6), the visual nature of the LO (3), and that it 
works best as a supplement to classroom teaching (2). 
Year 4 students also appreciated how the LO fit with 
the role of technology in their lives (6) but were more 
likely to stress that it should be a supplement to 
classroom teaching (4). They also suggested it provides 
good practice (3), although one said that it offered 
nothing new (1). Overall, most students were positive, 
citing their ease and familiarity with technology and 
valuing the option of hands-on practice. 

We run and depend on technology now so it is very 
desirable, if we can sit in our own room and learn 
I’ll take it. (Year 4 seminar) 

...the multimedia is more captivating than just a 
textbook. (Year 4 seminar) 

I think it is desirable because every student knows 
how to use a computer and you made it easy 
enough to use this learning object so I think 
everyone learned something for this assignment. 
(Year 2 Methods) 

...it is a different way of learning, and more hands- 
on, which allows for better understanding. (Year 2 
Methods) 

The more critical responses to this question help 
explain the slippage in scores on the questionnaire 
items about ‘value added.’ Five of the Year 4 Seminar 
students registered concerns that the LO, while a “great 
complement’’ to classroom teaching, “should not take 
the place of in-class learning” (Year 4 Seminar). The 
slightly higher quantitative scores by the Year 2 
Methods students may be explained by the fact that an 
instructor was present when they used the LO. Indeed, 
two of the Year 2 Methods students wrote that a 
limitation of the LO is that it cannot answer their 
questions in the way that an instructor can. One 
summed up this position by stating, “I think it is 
desirable to use technology as a supplemental teaching 
tool. Lecture and teaching assistant help holds its value 
as well because you can ask questions and get insight” 
(Year 2 Methods). 

Discussion and Conclusions 


On the whole, both groups of students showed 
enthusiasm for the qualitative coding LO, but as Kay 
and Knaack (2007) observe, many evaluation studies 
similarly report that LOs are well received. The 
important question is, did the LO help students learn 
what we wanted them to learn? Based on the strongly 
positive student self-reports of learning value, we think 
it did. Moreover, because practice with the LO was 
closely linked to the completion of coding assignments 
which were evaluated, students were in an excellent 
position to comment on whether the LO benefited their 
learning. This is especially true of the Year 4 Seminar 
students who completed the questionnaire after they 
had received feedback on their coding assignment. 

Another question that arises is why the 
questionnaire items to do with ‘value added’ — how 
well the LO augmented other forms of instruction — 
received lower scores. Gauging from the written 
responses, this section of the questionnaire taps into 
some students’ concern that the introduction of 
instructional technology in general may be advancing 
administrative cost-cutting agendas in the context of 
diminishing resources for undergraduate education. 
Although a majority of students did agree that the LO 
helped them in ways that traditional formats did not, the 
large number of tentative responses suggests that where 
a LO is not supported by and integrated within other 
forms of classroom instruction, students will respond 
more negatively to it. 

The question that remains is whether the LO was a 
useful medium for teaching students about coding 
considering the challenges we identified at the outset. It 
turned out that introducing such a complex process as 
qualitative data analysis was a lot to ask of a LO, and 
we were aware of the compromises to both content and 
process that had to be made. For example, our three 
sleuths demonstrate coding using only a single page 
excerpt of a single transcript. Students are told that the 
transcript is one of thirty, but they are not given a 
demonstration of the fact that most qualitative 
researchers work intensively with much more data. It is 
difficult to conceive how a LO could demonstrate (and 
allow students to practice) common qualitative data 
analysis techniques such as the Glaser and Strauss’ 
(1967) method of ‘constant comparison’ whereby 
researchers confirm the soundness of their coding 
categories by looking for similarities and differences 
across instances. 

Another area of qualitative data analysis that is not 
addressed by our learning object is how to do narrative 
analysis. As distinctive as our sleuths are, their 
approaches are all loosely based on grounded theory 
and qualitative content analysis. Narrative analysis is a 
much different approach which works with the whole 
transcript and avoids fragmenting the data by breaking 
it into categories. Narrative analysis gives primacy to 
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identifying how the research subject works to construct 
a narrative with certain properties and implications for 
the self. In contrast, our sleuths tend to give primacy to 
the substantive content of the interview. Finally, the LO 
does not demonstrate how to analyze the relationship of 
the interviewer and respondent, including the power 
dynamics between them. Our sleuths do not give much 
attention to the researcher’s questions or the research 
context, whereas many qualitative researchers examine 
both the role of the question and the questioner in the 
kind of data that get produced. 

Despite these omissions, the LO did succeed in 
modeling multiple styles and layers of qualitative 
analysis, and it provided students with an opportunity to 
grasp the interpretive nature of the process and to 
practice their own coding techniques and style in a risk- 
free context. Only after publishing the LO in two large 
learning object repositories did we realize just how 
unusual our project was in this respect. We have 
identified only one other institution at which learning 
objects were created for qualitative research instruction 
(Chcnail et ah, 2006). Moreover, a search of the 20,292 
online teaching and learning materials indexed in 
MERLOT (Multimedia Educational Resource for 
Learning and Online Teaching) at the time of writing 
revealed that “Sleuthing the Layered Text: Investigating 
Coding” is only one of two resources available on the 
topic of qualitative methods; the other is an on-line 
textbook. 

So, the learning object did prove to be a 
sufficiently flexible medium to meet our pedagogical 
objectives. It was flexible also in the way it served 
students and instructors in different types of courses. As 
a value-added tool to their in-class learning, students 
reported it as having merit. However, students were 
also clear that they wanted the LO to be a supplement 
to, and not a replacement of, their current instructional 
formats of lectures, seminars, and labs, which is a 
concern we also share. 
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